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ABSTRACT 

The influence of three types of evaluation 
information was studied in simulated decision-making situations 
within a community mental health center setting* Administrators 
reviewed decision problems and were then presented political^ 
cost/benefit^ and statistical information one at a time. After each 
presentation^ subjects rated the ioportanca they placed on the 
different types of information and then indicated their dacisions 
regarding the problem. The results indicated that only cost/benefit 
data significantly influenced administrators' decisions^ The effects 
of various ty^s of information were not dependent on the type of 
decision situation. Length of time in administrative positions did 
not significantly correlate with the level of importance placed on 
political^ cost/benefit, or statistical data* The findings indicated 
that if program evaluators are to have an impact cn decision-making, 
they must provide better cost/benefit data ana face up to a much 
tougher payoff^based evaluation strategy* (Author) 
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Abstract 



The Influence of three types of evalyation infonnation v^as studied in 
siiiiulated decision-making situations vnthi^ a cornmunity mental heaUh center 
setting. Administrators revlevved decision problems and were then presented 
political cost/benefit and statistical information one at a time. After each 
presentation, subjects rated the importance they placed upon the different types 
of information and then indicated their decisions regarding the problem. The 
results indicated that only cost/benefit data significantly Influenced administra- 
tors* decisions. The effects of various typos of inforniation were not dependent 
upon the ^pe of decision situation. Length of time in administrative positions 
did not significantly correlate with the level of importance placed on political^ 
cost/benefit or statistical data. The findings indicated that if program evalu- 
ators are to have an impact upon decision makings they must provide better 
cost/benefit data and face up to a jnuch tougher pay-off based evailuatlon strategy. 
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Cofiiparatl ve Influences of PolitfcaU Cost/Benefit and Statistical 
Evaluation Information on Adrnini strati ve Decision Making 

A niajor problem for an evaluatDr is tO' bo ablo to predict what types of 
infonTiation will be neaded and most useful to decision makers. V/hile program 
evaluators v/ould sometimes prafer to be final judges (Scrivens 1967) s they often 
function as advisors rather than decision mafcers within an organization (Weiss 5 
1973)- An evaluator has the responsibility to present inform'ation to decision 
makers in a form they can use and understand. When evaluation Information does 
not satisfy a decision maker^s needSs the information v/ill have little impact 
upon the decision-making process. 

The purpose of this study v/as to investigate the influence of different 
types of evaluation information on administrative decision making In community 
mental health centers. Two decision situations, a direct service problem and 
an indirect service problem, were simulated in a group setting with paper and 
pencil mat'orials* 

Tfie primary research hypothesis v/as that political, cost/benefit and statis= 
tical evaluation information has significafitly different levels of influence upon 
j:l ii! ii ! ^ i.r\i t ; vu dacision:;. in adciitlOfu i t was spdculatud that docision niaking 
based upon different types of information is dependent upon the type of decision 
situation and the length of time administrators have been in mental health program 
administrative positions. 
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Subject?^. Tv/elve riCMital Jicialtii center adniini strators viere randomly 
iinslgnof! to one oF twu decision-iiiiking situations. The subjects v/ere asked to 
aS5ui?Li tlio rolo of the i jntal health center director in each decision situation. 
A limited ainount of detail about the decision problem was initially presented 
so that subjocts would be compelled at first to rely upon their ov/n personal 
experiences and IniprGSsions and tfien the additional evaluation information as 
it v/as presentecl. 

no^is\Qn Situations, A parallel fornis format was used in the two decision 
situations. In each simulation tfie situation and background information v/as 
presented^ follov/ed by each type of additional Inforination one at a time. 

In th.' di rect services situation the niental health center director must 
decide liow many hospital beds v/ill be needed during the coming year for mental 
health services. It must also bo decided v/hether or not additional beds should be 
reserved for alcohol detoxification services. The center direct renegotiates a 
contract on a yearly basis with the local hospital for in-patient mental health 
services. A flat annual fee for administrative overhead is charged by the hospital 
according to the nuniber of beds to be reserved and available. 

The indirect services decision required the center director to decide how 
many dnllars to budciot during the coming year for mental health education and con- 
sultation services. Since a federal staffing grant drops to a minimum 30 porcent 
funrliiir] invfii n^vt yon^", the dir^cctor v/ill need to ask cnunty commissioners for 
more money to continue tt^ese indirect service programs. The center hab had a 
full-time mental health educator the past four years. The educator is getting 
married next month and plans to resign. Three other full-^tlme staff have been 
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working half of thGir tiiiiO in coinfiiunity consultation with schools^ church groups, 
and lav/ enforcement agsncies. The center director must decide if he should decrease 
the 42,800 dollar consul tatian and education budget or expand and possibly reorganize 
the CDnstil tation and education progranu 

IjlM tjii erU ion . Three types of information v/ere presonted to the ad- 
ministrators for each decision situation. The same types of data v/ere prepared 
for both decision-making situations, but the content of the tv/o parallel forms of 
infoniuition v/as different. The format^ persuasive composition ^ and amount of infor- 
mation v/£jre the same for both decision situations. 

F^olitical information included the opinions of internal mental health center 
staff, the center's external advisory council of lay person representatives, and 
the existing leanings of local county commissioners wlio comprise the center's 
governing board. The pov/er structure at both the state and federal levels was 
discussed. It v/as an election year, and the political considerations and conse^ 
quences of the decision options v^ere reviev/ed. 

Cost/Benefit infoniiation was based upon the Program Planning and Budgeting 
System (PPDS). Last yearns planned program objectives, program budgets, and pro- 
gVani uutcoiM'S v/ere presonted. The staff v/ere moderately successful in attaining 
program objectives, but actual costs exceeded projected unit benefits for last 
year. Whilo the federal staff grant support is declining^ local funding have 
alv/ays previously been approved to pick up the annual reductions in federal funds. 

Statistical data was pro^rMitnd on the catnhinont area residents fininr] served. 
The population is predominantly rural with only tv/o large u. *'an centers. A sig^- 
nificanL portion of the population is non--v/h1te. The actual ,:iiization of the 
respectivo services was also suipmarized. An Inference v/as made regarding the 
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porcnntaor: of people in the catchniant area who could benefit from services » i,e, 
the c mnd or need. The existing levol of utilization (viz, met der iiand } was then 
subtracted from the total infer rod demand, and the difference was the projected 
j mrnst ne_ed^ 

D^^j^ ii::gn Ul prQcodu ros , Twelve buolclets of siniulatfon material v/ere ran- 
domly distributed at a group iiieoting of coinmunity mental health center adrniniS'- 
trators. Six of the booklets v/ere siniulations of the direct services situation, 
and six were simulations of the indirect services situation. Each booklet included 
a general subject background information sheet, the respective decision situation, 
four decision-rnaker opinionnai res and the three sets of evaluation information. 
Subjects v/ore instructed to assume the role of the community mental health center 
director, read through the booklet and respond to the decision-making opinionnaires 
as they were presented. 

Lxporinen tal design. The order of presentation for the three sets of evalu- 
ation inforniation v/as counterbalanced v/ithin each experimental group. The decision 
maker opinionnai re was presented immediately following the simulation situation and 
again after each type of additional evaluation information was presented. The 
overall dasigfi was, thus, 3 4x2 factorial with repeated measures In both treatment 
groups, 

Th^^ 5^^ig.jAloiLi^^ker op i n i onn a i re . The assessment instrument was a 10 item 
Likoj^t Scalo which measured administrators' opinions regarding the Importance and 
ui^eru!/,- ■>i; u. puliLical, cost/benefit and stuListical data. In addition to the 
10 LikurL item, two questions v/ero posed regarding what the administrator would 
docido givon the existing amount of infoniia t1 on, 
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The Likort Scale response itcnis v^ere quantified on a 10 to 70 scalu t:h 
40 being the scale midpoint. Individual iteiTis vvu coded so that the greater 
the level of importance placed upon the evaluation Information^ the larger the 
subject's nunieric score. 

ResuUs 

InFluence of JjifprniSiioib- The effects of the type of information presented 
and the type of decision situation confronted are summarized in Table 1, The 
treatiiient effect of type of evaluation information was statistically significant 
(Fj 30 ^ 5.38; p<.001) v/ith no interaction effects betv/een type of decision and 
type of in Formation presented. 

Table 1 

Decision-Maker Opinionnaire Differences With 
Repeated Measures on Type of Information 



Source SS df MS 



B^tA'/e eMi_s u b j e£t s 

Typo of Decision (B) 6533.33 1 6533.33 1.30 

Subjects within Groups 5(B) 50108.25 10 5010,82 
lii t^ri n subjejcts 

Type of Information (A) 29541.66 3 9847,22 5.33* 

AD 5049.97 3 1683,32 .92 

AS (li) 54956.59 30 1831.89 

Total 146189,80 47 



*p X . Ul 
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In ordor to identify which tynci of evaluation information significantly 
influoiiCed adniinistrators ' decisions, the Tukey T-method was used to make pairv/ise 
comparisons betv/een the pretest control and post-^treatment assessnients. 

Results of the £05 jt hoc analysis are shown 1n Table 2. 

Table 2 

Comparisons Betweon Means of Different 
Types of Evaluation Information 



Difference in Group Means 



Group "X Control Political Cost/Benefit Statistical 



Control 476,67 36.66 69.16* 25.83 

Political 32.50 10.83 

Cost/Bonefit — 43.33 
Statistical 

*p < .05 

The difference in mean decision-maker oplnlonnaire scores was statistically 
signiflcafit for the cost/l)enef i t evaluation inforniatlon versus no evaluation in- 
forination assessments. No otfier pairwise comparisons v/ere significant at the .05 
level , 

The nU'iibor of months of cidministrative oxporience for each subject was corrolated 
v/i th the Initial level of im[)0rtance5 I.e* crntrol assessment, the subject placed 
upon political, cost/benefi t , and statistical information. The findings from the 
three respoctlve Pcarsofi product-moment correlations are presented in Table 3. 
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Tabic 3 



Relationship Butwaen Adnilnistrative Exparience and 
Iinportance of Types of Information^ 



Type Inf Qrriiation 



Pol i ti cal 



-.17 



Cost/Benefit 



.12 



Statistical 



-.12 



Noto. Administrative experience v/as nieasured in months. 



a 



n - 12 for each type of information. 



Review of Table 3 indicates no significant relationship betv/een administra'- 
tive experience and importance initially placed upon different types of data. 
In facts a negative correlation v/as found for political and statistical data* That 
is, tfie greater an administrator's experience^ the less importance they placed 
upon political and statistical information. 



The results support the contention that administrators respond diffey^ently to 
various types of evaluation information. More specifically, only cost/benefit 
data was significantly influential in shaping administrative decisions for the 
two simulated decision situations. 

The overall findings did not indicate a relationship necessarily exists between 
type of decision to be niade or length of administrative expo^ience and the importance 
placed on various types of evaluation data. The opinionnaire data revealed^ hov/everp 
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that cost/benorit information is more important than other types of data to 
administrators irregardless of their professional adniinistrati ve experiences. 
This is especiany meaning Ful given the fact three fourths of the respondents 
(n - 9) had at least thy^ee years of clinical as well as administrative mental 
heal til experi once -. 

This study supports remarks by Scrivon (Salasin, 1974) that evaluators must 
collect better cost/benefit data and face up to a much tougher kind of pay-off 
based evaluation strategy. Furthermore, the study shows that administrators 
believe cost/benefit data are more important in decision making than either statis- 
tical or political Information, 

Administrators are not certain about the role of an eyaluator. They do not 
agree on the competencies needed by evaluators to mruu their needs, Evaluatov^s must 
attend to what administrators need and feel is important information. The results 
and usefulness of the information evaluators provide will determine if evaluation 
is to play a significant role in the decision-making process. 

Current evaluation models often stress inappropriate strategies for evaluators. 
Measuroniont-based strategies need to be replaced by policy-making models. At the 
same time evaluators have not been the final judges in decisloti making. Expert 
opinion cannot overrule economic constraints and political demands. Evaluators must 
be trdincd to recognize these limitations and collect information which reflects 
and hopofully will satisfy economic qu-^ tions. 

L v'ci 1 uj Luri li.ay viov; tiiciijSQ 1 ves an inrorination provider's (Stufflebeam et, al,, 
19/1) or as individuals v/ho can facilitate judgments about the social and economic 
worth of huinan service programs (Scriven^ 1972; Stake, 1972; l/eiss, 1973). But 
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tho overriding conclusion from this study is that evaluators may have little 
or no impact upon decision making unless they can provldG inforn;ut1on to 
decision makers v/hich decision makers believe is important and useful. 

Evaluators must continually ask themselves: Is the infonnation I am collecting 
not only ini()ortant to me^ but also important for the person who v/ill make final 
decisions? 
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Roference Note 

1, Stake, R.E, An approach to the evaluation of instructional programs 
(Program portrayal vs, analysis) Paper presented at annual meeting of 
American Educational Research Association^ Chicago^ minois, 1972, 
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